Extensible framework for executable annotations in electronic content

ABSTRACT

Insertion of an executable command or instructions in the form of an annotation to electronic content, such that the receiver can benefit from a more precise and detailed context as they interpret and/or act on the sender&#39;s request. The system can, upon detecting an appropriate trigger, be configured to ‘translate’ or convert the annotation across multiple applications and platforms and perform one or more tasks that facilitate the receiver&#39;s access to and interaction with the item. The sender enjoys a much finer control over the manner, presentation, and overall context of the electronic content item as it is shared, and becomes able to offer the receiver a more precise understanding of their own intentions in sharing this item. Similarly, a receiver can be guided toward actions or responses that are expected or desired by the receipt of the electronic content item.

BACKGROUND

Computing devices that include cameras and multimedia content supportapplications have become increasingly common in mobile devices,including laptop computers, tablets, digital cameras, smartphones, aswell as other mobile data, messaging, and/or communication devices.Generally, users make use of cameras associated with computing devicesto take various pictures, such as images of scenery, persons,presentations, whiteboards, business cards, documents, sketches,paintings, and so forth. The users can refer to the captured images torecall information contained therein (e.g., diagrams, pictures, listsand other text) and/or to electronically deliver them to other users,storage services, or devices.

Multimedia content is typically shared among mobile users throughmultimedia messaging services (MMS), and multimedia content is sharedbetween Internet users and mobile users via email. However, the sharingof multimedia content stored in the device (e.g., photos, video clips,audio clips, data, etc.) with friends, colleagues and other familymembers remains relatively nuanced and cumbersome. Users may havespecific intentions associated with the sharing of the content that arelost during the transmission, or that require a time-intensive effort toconvey. Thus, there remain significant areas for new and improved ideasfor the efficient and meaningful sharing of electronic content, as wellas management of the type of responses that are desired from therecipients.

SUMMARY

A data processing system, in accordance with a first aspect of thisdisclosure, includes at least one processor and one or more computerreadable media. The computer readable media include instructions which,when executed by the at least one processor, cause the at least oneprocessor to present over a communication network, on a first clientdevice, a first user interface for a first content viewing application,the first user interface displaying an option for associating a firsttag with a first electronic content, as well as receive over thecommunication network, from a first user, a first user input forselecting the option to associate the first tag with the firstelectronic content, the first tag representing a first executableprocess for performing a first task. In addition, the instructions alsocause the at least one processor to record, in response to receiving thefirst user input, an association between the first electronic contentand the first tag. Furthermore, the instructions cause the a least oneprocessor to transmit over the communication network the firstelectronic content to the first recipient, and execute, in response totransmitting the first electronic content, the first executable process.

A method executed on one or more computing devices, in accordance with asecond aspect of this disclosure, includes presenting over thecommunication network, on a first client device, a first user interfacefor a first content viewing application, the first user interfacedisplaying an option for associating a first tag with a first electroniccontent. The method also includes receiving over the communicationnetwork, from a first user, a first user input for selecting the optionto associate the first tag with the first electronic content, the firsttag representing a first executable process for performing a first task.In addition, the method includes recording, in response to receiving thefirst user input, an association between the first electronic contentand the first tag. Furthermore, the method includes transmitting overthe communication network the first electronic content to the firstrecipient, and executing, in response to transmitting the firstelectronic content, the first executable process.

A data processing system, in accordance with a third aspect of thisdisclosure, includes means for presenting over the communicationnetwork, on a first client device, a first user interface for a firstcontent viewing application, the first user interface displaying anoption for associating a first tag with a first electronic content. Thesystem also includes means for receiving over the communication network,from a first user, a first user input for selecting the option toassociate the first tag with the first electronic content, the first tagrepresenting a first executable process for performing a first task. Inaddition, the system includes means for recording, in response toreceiving the first user input, an association between the firstelectronic content and the first tag. The system further includes meansfor transmitting over the communication network the first electroniccontent to the first recipient, as well as means for executing, inresponse to transmitting the first electronic content, the firstexecutable process.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations in accord withthe present teachings, by way of example only, not by way of limitation.In the figures, like reference numerals refer to the same or similarelements. Furthermore, it should be understood that the drawings are notnecessarily to scale.

FIG. 1 illustrates an implementation of an image sharing and annotatingapplication and environment;

FIG. 2 is a conceptual diagram illustrating one implementation of adistributed computing environment for enabling executable annotationswith electronic content items;

FIG. 3 is a display diagram illustrating an implementation of a userinterface for an application configured to provide annotative tools forelectronic content items;

FIG. 4 is a display diagram illustrating an implementation of a userinterface for a communications application conveying an augmentedelectronic content item to a recipient;

FIG. 5 is a display diagram illustrating an implementation of a userinterface for a communications application presenting a response to aninteraction event by the recipient;

FIG. 6 is a display diagram illustrating an implementation of a userinterface for payment processing application;

FIG. 7 is an illustration depicting an instant messaging applicationpresenting an automatic notification informing a sender that a purchasehas occurred and/or providing details about the transaction;

FIGS. 8A-8F are a sequence of illustrations depicting an implementationof an image sharing interface and an example of an execution ofinstructions tagged with the image as it is shared;

FIG. 9 is a display diagram illustrating an implementation of a userinterface for an application configured to provide annotative tools forelectronic content items;

FIG. 10 is a display diagram illustrating an implementation of a userinterface following execution of instructions conveyed via the augmentedelectronic content item of FIG. 9;

FIG. 11 is a display diagram illustrating an implementation of a userinterface for an application configured to receive status informationrelated to the electronic content item of FIG. 9;

FIG. 12 is a process diagram of an implementation for user customizationof annotative tools;

FIG. 13 is a flow diagram illustrating an implementation of a processfor sharing and managing annotated electronic content;

FIG. 14 is a block diagram of an example computing device, which may beused to provide implementations of the mechanisms described herein; and

FIG. 15 is a block diagram illustrating components of an example machineconfigured to read instructions from a machine-readable medium.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent that the presentteachings may be practiced without such details. In other instances,well known methods, procedures, components, and/or circuitry have beendescribed at a relatively high-level, without detail, in order to avoidunnecessarily obscuring aspects of the present teachings.

The following implementations introduce a scan application toolboxconfigured to enhance user content sharing experiences by permitting auser to add an executable label or annotation to the content prior tosharing. Traditionally, content sharing workflows have included a seriesof cumbersome steps as a user attempts to share their electronic contentwith a specific intent of the mechanism by which it should be conveyedor to whom or a desire that further action to be taken or initiated bythe recipient. The sender typically must (1) identify the item; (2)insert the item into an application for sharing; (3) add some note ormessage that indicates the response or result desired by the receiver ofthe shared item; (4) hope that the receiver understands the request andtakes the steps necessary to respond appropriately. Such notes oftenremain embedded or visibly printed onto or alongside the electroniccontent, which may not be the preferred outcome for the sender, whoadded the annotation to ensure the receiver acted on the information asrequested. The disclosed implementations allow a user to add anexecutable command or instructions in the form of an annotation to theelectronic content, such that the receiver can benefit from a moreprecise and detailed context as they interpret and/or act on thesender's request. In some implementations, the system can be configuredto ‘translate’ or convert the annotation upon detection of a predefinedtrigger and perform one or more tasks that will facilitate thereceiver's access to and interaction with the item. The sender enjoys amuch finer control over the manner, presentation, and overall context ofthe electronic content item as it is shared, enabling them to offer thereceiver a more precise understanding of their intentions in sharing theitem. Similarly, a receiver can be guided toward actions or responsesthat are expected or desired by the sender of the electronic contentitem, while also finding that the process of responding or engaging withthe item greatly simplified.

As introduced above, applications such as word processors, publishers,spreadsheets, presentation software, and others can be used to generateelectronic documents or content. Generally, the term “electronicdocument” or “document” includes any data that may be presented (e.g.,visually, audibly, tactile-based, etc. presented), including but notlimited to an electronic content item and portions thereof, a mediaitem, a media stream, a web page, a hypertext document, an image,digital video or a video recording, digital audio or an audio recording,animation, a markup language document, such as a HyperText MarkupLanguage (HTML) or eXtensible Markup Language (XML) document, a formhaving blank components to accept entered data, or data describing theapplication of a GUI, image documents that include images of text thatmay be extracted using optical character recognition (OCR) processes,documents that may include mixtures of text and images, such as PortableDocument Format (PDF) documents or PowerPoint (PPT) documents, etc., orany type or format of document from which text may be extracted or thatmay be converted to text, and other digital data. As an example, theelectronic content may include word processing documents, spreadsheets,presentations, e-books, images, slides, photographs, or otherdigital-based media.

In addition, a user or participant generally refers to one who views,creates, captures, modifies, develops, collaborates, suggests, listens,receives, shares, reviews, revises, or disseminates pieces of electroniccontent, including the creation, viewing, or updating of annotationsassociated with the electronic content. A user includes a creator orsharer of electronic content and electronic content-based programs, aswell as a user of the apparatus and systems described herein.Furthermore, the term “software application”, “software”, or“application” refers to a computer program that performs useful work,generally unrelated to the computer itself. Some non-limiting examplesof software applications include speech-to-text applications, imagecapture applications, recording applications, word processors,spreadsheets, slideshows, and presentation design applications,accounting systems, and telecommunication programs, as well as gamingsoftware, utility and productivity tools, mobile applications,presentation graphics, and other electronic content creation or sharingsoftware.

In different implementations, software applications such as programsoffered in the Microsoft Office Suite® (e.g., Word®, Powerpoint®,Excel®, Visio®, Skype®, Teams®, OneNote®), Office 365®, Microsoft 365®,Microsoft Pix®, Microsoft Office Lens® and other suites or applicationscan offer a variety of teleconferencing, text generation, presentation,and image capture or commenting tools. In other implementations, AdobePhotoshop®, Google Slides®, or another image capture or datapresentation program, such as Clear Scanner®, Adobe Scan®, CamScanner®,Apple Pages®, Corel WordPerfect®, Google Docs®, IBM Lotus Word Pro® orother software applications within the Microsoft Office Suite® or arrayof Microsoft® products as well as any non-Microsoft® based applicationsmay be used. These are non-limiting examples, and any other electroniccontent creation, editing, or collaboration application may benefit fromthe disclosed implementations. Throughout this description, the softwareapplication that is used to originally create or normally view (e.g., ona desktop) the electronic content will be referred to as the nativeapplication.

In some implementations, various image sharing and scanning applicationspermit end-users to scan documents, presentations, real-world objects,and other subjects using images captured by a camera associated with thedevice or via images stored or accessed from memory. In differentimplementations, software applications such as programs offered in theMicrosoft Office Suite® (e.g., Office Lens®, Powerpoint®, Visio®) andother applications can offer a variety of image capturing and editingtools, including scanning and identification of different regions in animage. Other examples include Microsoft Safety Scanner®, VueScan®,Picasa®, TWAIN®, Windows Fax and Scan®, PaperPort®, SilverFast®, GeniusScan®, TurboScan®, Scanner Pro®, Prizmo®, Google PhotoScans® and HelmutFilm Scanner®, Google Drive®, Evernote Scannable®, Dropbox®, Scanbot®,CamScanner®, Photomyne®; these are non-limiting examples, and any otherelectronic content editing or viewing application may benefit from thedisclosed implementations.

The software application that may incorporate the disclosed features canbe installed on a client's device, or be associated with a third-partyapplication, such as a web-browser application that is configured tocommunicate with the device. These devices can include, for example,desktop computers, mobile computers, mobile communications devices (suchas mobile phones, smart phones, tablets, etc.), smart televisions,gaming devices, set-top boxes, and/or any other computing devices thatinclude a camera and/or an image-display capability.

In order to better introduce the systems and methods to the reader, FIG.1 presents an example of a representative electronic content sharingenvironment for implementing an executable tagging or annotation feature(the system is illustrated schematically in greater detail in FIG. 2).In different implementations, the environment can include a plurality ofcomputing device end-users, or simply “users” who can capture, view,edit, modify, and/or interact with the electronic content (for example afirst user, a second user, a third user, etc.). For purposes ofsimplicity, in the following scenarios a user who is responsible foradding, attaching, linking, inserting, or otherwise associating anexecutable annotation to an electronic content item will be referred toas a sender, and users, groups, or entities who receive or are otherwiseaffected by the annotated electronic content and/or any manifestation ofthe executed annotation will be referred to as recipients. Thus, arecipient need not be limited to persons or individual communicationaddresses or contact points, but may be understood to more broadlyencompass any sharepoint, including e-mail accounts, instant messagecommunications, text messages, messages transmitted via teleconferencingapplications, social media pages for individuals and/or groups,collaborative applications, websites or organizational sites, cloud ornetwork storage, or any other mechanism by which the electronic contentmay be forwarded, shared, moved, or uploaded from a first digitallocation to a second digital location. In some implementations,transmission of a file or moving of the file from one location toanother can be configured to trigger execution of the instructionscarried by the annotation.

As will be described in further detail below, an executable annotationcan be understood to refer to any virtual note, label, tag, orinformation that is distinct to the data comprising the electroniccontent item itself and is paired with or otherwise includes executableinstructions or code for performing one or more tasks. Executableinstructions can be configured to cause a computing or other processingdevice to perform a certain function or group of functions. In someimplementations, the executable instructions can be executed by acomputing system that is associated with a sender and/or targetrecipient, while in other implementations the executable instructionscan be executed via a cloud computing system or network.

As an example, a sender 102, a first recipient 110, a second recipient120, and a third recipient 130 are depicted in FIG. 1. In this case, thesender 102 is accessing an electronic content item (here, an image 104)on a mobile computing device 106 that is linked via a network to otherend-users, data repositories, and entities. As the user views the image104, he or she can add or otherwise associate one or more executableannotations to the image 104. For purposes of simplicity, the term ‘tag’may be used interchangeably with the term ‘executable annotation’ andthe act of ‘tagging’ will also refer to the process of associating anexecutable annotation with an electronic content item.

In some implementations, the selection of a tag can occur via a userinterface 108 presented on a display. In general, an “interface” can beunderstood to refer to a mechanism for communicating content through aclient application to an application user. For example, interfaces mayinclude pop-up windows that may be presented to a user via nativeapplication user interfaces (UIs), controls, actuatable interfaces,interactive buttons or other objects that may be shown to a user throughnative application UIs, as well as mechanisms that are native to aparticular application for presenting associated content with thosenative controls. Furthermore, an “actuation” or “actuation event” refersto an event (or specific sequence of events) associated with aparticular input or use of an application via an interface, which cantrigger a change in the display of the application.

In addition, a “native control” refers to a mechanism for communicatingcontent through a client application to an application user. Forexample, native controls may include actuatable or selectable options or“buttons” that may be presented to a user via native application UIs,touch-screen access points, menus items, or other objects that may beshown to a user through native application UIs, segments of a largerinterface, as well as mechanisms that are native to a particularapplication for presenting associated content with those nativecontrols. The term “asset” refers to content that may be presented inassociation with a native control in a native application. Thus, asnon-limiting examples, an asset may include text in an actuatable pop-upwindow, audio associated with the interactive click of a button or othernative application object, video associated with a teaching userinterface, or other such information presentation.

In the example of FIG. 1, the sender 102 has selected and/or createdthree different tags, including a first tag 152, a second tag 154, and athird tag 156. In different implementations, a tag may contain, include,or convey tag content. As one example, the tag content of the first tag152 includes an ‘at’ mention to an individual user account as well as anote (“@Jason—Is this flower listed?”). In some implementations, theformat and/or use of a specific symbol or character can determine ordefine the dynamics for a particular task that will be performed whenthe tag is executed. For example, the use of an “@” followed by anidentifier can indicate that the electronic content item should beshared with the named user in a particular way. In addition, the use ofa dash “-” (or other symbols, such as a colon, comma, quotes, hash,etc.) can indicate that the text that follows the colon should bepresented as a message to the named user. These details are shown forpurposes of illustration only, and it should be understood that a widerange of other tag content can represent or serve to convey differenttypes of instructions.

The tag content for the second tag 154 includes a plus sign (“+”)followed by a file name (here “Flowers.PPT”, where the powerpoint fileextension suggests the file is a slideshow). In this example, the plussign can indicate a request that the image 104 be shared or otherwiseadded to the named file. Again, this usage is for purposes ofillustration only, and the particular formatting or type of charactersused to express any instructions can vary. In a third example, the tagcontent for the third tag 156 includes “#MarylandPlants”, indicating arequest by the user for the image to be shared on a social media webpagelinked to the MarylandPlants identifier.

It can be understood that each tag has been inserted or associated withthe image 104. In different implementations, once a tag has beenassociated with an electronic content item and the tagged electroniccontent item is shared or otherwise moved, the system can be configuredto execute the instructions paired with or contained within the selectedtag. A user may, in some implementations, access or otherwise manage thetagged image 104 via a first application. It can be appreciated thatwhile execution of the task(s) associated with the tag can be triggeredduring use of the first application (e.g., the user takes some actionrelative to the image 104 that triggers the executable instructions),the executed task may also occur via the first application and/orthrough the resources of a second, different application. In otherwords, the tag can be configured to perform a wide range of task typesacross a variety of platforms. Thus, the task requested by the insertionof the tag need not be limited in scope to what the first application iscapable of, but can extend across multiple applications andfunctionalities.

In FIG. 1, for purposes of simplicity, the sender 102 submits or uploadsthe tagged image to an executable annotated content manager application(“manager application”) 160 that is configured to access a cloud-basedannotation system via a network connection. In other implementations,the manager application 160 may be optional, and its functions carriedout by the software that is used to create, modify, view and/or sharethe electronic content item, or a user may directly access the systemonline as a cloud-based application. Upon detecting and/or identifying asubmission of tagged content, the manager application 160 can beconfigured to process any tags and/or to forward the tagged image toanother module or application for execution or presentation. In thiscase, the manager application 160 identifies and classifies each tag.Depending on the type of tag and its corresponding task, the managerapplication 160 can route the tagged content to the appropriate outletor executing application.

In a first scenario 170, upon execution of the first tag 152, the image104 has been transmitted in a message 172 to an e-mail inbox for thefirst recipient 110, where the message subject line includes the sametext of the note “Is this flower listed?” In a second scenario 180, uponexecution of the second tag 154, the image 104 has been added to andembedded in the file “Flowers.PPT” 182, where the selected file may beunderstood to serve as the second recipient 120. In a third scenario190, the image 104 has been posted to the designated third recipient130, here a social media page 192. Thus, it can be appreciated that indifferent implementations, each tag can be paired with a distinctlydifferent task. When the instructions are executed for each of thesetags, multiple processes are initiated across several applications. Inaddition, in some implementations, some or all of the steps comprisingthese tasks are performed automatically, without further userintervention or inputs. It may be appreciated that this type of systemcan offer users an improved experience as they create and share contentacross multiple distribution paths.

Referring now to FIG. 2, an example of a representative architecture fora content annotation and sharing management system (“system”) 200 aredepicted. In different implementations, the proposed systems can includeprovisions for applying a variety of annotation-task pair types toelectronic content and/or presenting various user interfaces for tagcustomization. In some implementations, the user experience of insertingan annotation may be relatively straightforward. In FIG. 2, an exampleof a system that is configured to manage the use of executableannotations is illustrated. It is to be understood that the system 200presented here is merely an example implementation, only some aspectsare presented for purposes of clarity, and that a wide variety of otherimplementations are possible.

While the system 200 can be installed locally on a client device, inother implementations, as shown herein, the system 200 is stored and/orconfigured for access via a network and/or cloud-based computingservice. In FIG. 2, it can be seen that the system 200 includes anannotation input component 210. The annotation input component 210 isconfigured to store, update, and provide or enable access to theplurality of annotation options available for insertion or associationwith an electronic content item. As an example, the annotation inputcomponent 210 can include a set of labels 212 (e.g., Label A, Label B,Label C, etc.) that may be selected for linking or pairing with one ormore specific corresponding tasks 214 (e.g., Task A, Task B, Task C,etc.). In some implementations, pairings can be established as a defaultby the system; for example, there can be pairings that are specific toand are expected to be commonly used by particular user-types such asbusinesses, vendors, social media, families, individuals, professionalor personal networks, travel, blogs, schools, or other groups. In otherimplementations, such pairings can occur as a result of customizeduser-designated or created label-task connections, via a customizationmodule 216.

A sender 202 accesses an electronic content item 204 via any type ofdevice 228 capable of presenting images and/or digital content, such ascameras, mobile phones, tablets, laptops, desktops, gaming devices,projectors, and other such devices. Furthermore, software such as acontent-sharing application can be locally installed on the device, orsome or all aspects or features of the application may be accessed fromanother device or accessed from cloud storage computing services.

In different implementations, the device 228 can access, store, orreceive electronic content item 204 for the sender 202 to view andannotate on a user device 228. In some implementations, the sender 202can attach or associate at least one label-task pair 218 to theelectronic content item 204 to produce an augmented electronic contentitem 206. The term augmented in this case can be understood to refer tothe result when a core content item (e.g., electronic content item 204)is associated with added features, content, attributes, or services suchas a tag. The additional features confer on the item a framework that isregistered during the sharing process. As one example, the sender 202can access label-task pair(s) from an online repository 290 a, and thelabel-task pair 218 can be updated or otherwise modified via a networkconnection 290 b to the system 200.

The user inputs can vary widely based on the type of input means used.In FIG. 2, the user input can be understood to be associated orcorrespond with some usage of a sharing application 208, including butnot limited to telecommunication software, storage synchronizationservices, social media platforms, and other such mechanisms for sharingcontent between one location and another. As one example, the user inputcan include a touch on a touch-screen interface or a mouse-clickdesignating a target recipient or location that is desired by the userfor sharing of the electronic content item 204.

In some implementations, the user input can also be identified as atrigger action event 226 by a trigger detection module 220 of theannotation management system 200. Generally, a “trigger event” or“trigger” refers to an event (or specific sequence of events) associatedwith a particular use of an application, which corresponds to aselection of an option offered via a native control, or an event thatmatches a condition. In FIG. 2, the triggering event may be understoodto include a ‘click’, toggle, voice command, or other input actions(such as a mouse left-button or right-button click, a touchscreen tap, aselection of data, or other input types) that leads to a sharing of theelectronic content item 204 or an activation of the augmented electroniccontent item 206. It can be seen that while the trigger action event 226occurs through a user device 228, the occurrence of the event may betransmitted by a network connection 290 c with the system 200.

Once a trigger is detected, the augmentation layer or layer of contentcorresponding to the annotation that was added to the electronic contentitem 204 can be processed by an annotation translation module 222, whichcan be configured to interpret the instructions conveyed by theannotation and determine the manner in which the instructions are to beexecuted. This information can then be provided to an annotationexecution module 230, which initiates execution via a designated taskperformance engine 232, and produces an outcome that is aligned with thesender's request; examples of this process are presented with referenceto FIGS. 3-11 below. For example, the annotation translation module 222can allocate a display area in the client device display, inside ofwhich a contextual layer can create user interface (UI) controls. Theannotation translation module 222 can execute instructions that renderthe contextual layer in the appropriate display area and adapt themanifestation of the task to the content of the contextual layer. Thedisplay area that is allocated depends on the layer's display mode. Inone embodiment, a layer cannot modify the size of its display area, movethe display area, or create UI elements outside of the display area. Inanother embodiment, a layer can request the properties of its displayenvironment (e.g., display mode, display area size, display arealocation, and pinning status) and/or be notified of any changes.

In some implementations, the outcome can include the transmission of theelectronic content item 204 within or alongside an augmented context234. In other words, the result of the execution of the instructions caninclude the presentation or delivery of the electronic content item 204to a recipient 252 (e.g., via a network connection 290 d) with asecondary layer of content that corresponds to the manner and/ormechanism by which the sender 202 requested the content should bemanifested for the recipient 252.

Furthermore, in different implementations, the annotation managementsystem 200 can be configured to guide or assist in the execution ofadditional actions stemming from the delivery of the augmented context234. For example, in response to various interaction events by therecipient 252 (e.g., via a network connection 290 e) with the receivedcontent, the system can manage subsequent actions. As a general matter,an “interaction event” as used herein refers to an event (or specificsequence of events) associated with a particular use or access of anelectronic content or application, which may then be used to determinewhether the additional actions should occur.

In FIG. 2, the recipient action 244 can be evaluated by a recipientinteraction workflow module 240. Depending on the type of action thatoccurs, the system can generate a reaction 242 that may communicate withexternal applications 246 and/or result in the system recording theaction in an activity log 260. Such information can be used to notifythe sender 202 (e.g., via a network connection 290 f) of the status ofthe executed annotation in some implementations, for example, via anauto-response generator 270.

As a general matter, a contextual layer generated by the annotation isexecuted as an extensible framework that may include the followingfeatures: (a) An XML file format for packaging a layer's HTML,JavaScript, and application descriptor into a single file. Theapplication descriptor indicates which types of electronic content thelayer is applicable to; (b) A service that renders a layer XML file intoHTML/JavaScript that can be displayed in a web browser; (c) Client-sideJavaScript for existing web applications (e.g., electronic contentprocessing modules) to instantiate layers and display them inside acontainer element. The JavaScript creates sandboxed iframe HTML elementsfor each layer. The iframes use uniform resource locators (URLs) thatpoint back to the service to render the requested layer; (d) Client-sideJavaScript for a remote procedure call (RPC) framework that enablesapplications to register JavaScript methods (the Layer API) that thelayers can call from their sandboxed iframes. This enables layers tocall back into the application and send/receive data. Furthermore, theremay be provisions for execution of the instructions while preventing thecontextual layer from interfering with the operation of the electroniccontent presentation. In some implementations, the annotation canindicate in its application descriptor which features it requires toexecute.

For purpose of clarity for the reader, FIGS. 3-11 present some examplesof user interfaces by which augmented electronic content sharing mayoccur. Referring first to FIG. 3, an implementation of a display 300 fora computing device 308 is illustrated. In different implementations, thedisplay 300 may present various icons, interfaces, graphics,applications, or other device status information. As an example, thedisplay 300 includes a representation of user access of a fileassociated with a first application. For purposes of simplicity, thefirst application in FIG. 3 is an image processing and presentationprogram that includes a viewing interface 310 configured to facilitatethe user's access to various documents or files via the application.Here the viewing interface 310 presents a first content item 312 for theuser to view or modify.

Once a user accesses an electronic content item, they may be able toaccess options or tools that can be used in association with the contentitem or application. As shown in FIG. 3, the first content item 312 isbeing presented in conjunction with a first annotation interface (“firstinterface”) 350 of an annotation management system, identified in thiscase by a header “TAGS”. In some implementations, the first interface350 can be separate from, or positioned or located adjacent or proximateto the viewing interface 310 of the first application, or may extendoutward from the viewing interface 310. In another implementation, thefirst interface 350 can be disposed within a region of the viewinginterface 310, and/or can be offered as a feature of the firstapplication itself. The first interface 350 can be configured to displayor present various options by which a user may select or customize a tagand/or associate the selected tag with the content item. These optionscan be presented independently through a tagging software application,or can be offered as a part of the features of the first applicationitself. For example, a user may choose to print, save, share, close,etc. the currently accessed item, as well as modify or associate one ormore of the tags to the item through a single application.

In some implementations, the first interface 350 can includenavigational tools or headings to assist a user in their efforts tonavigate the available annotations. In FIG. 3, a main header 352(“Vendor Tags”) is displayed, indicating that the user is currentlyviewing the tags classified or commonly used by customers interestedprimarily in transactional type services. In other words, the taglisting may be modified or requested to display those tags that arefrequently used by those who are involved in business exchanges,merchant tasks, and other product related interactions. In otherimplementations, the tags may be sorted or filtered by other means, or auser may customize the listing of tags that are shown. In this case, aplurality of annotation options 354 are shown, in which a first option356 (“BUY”) is selected. This tag, when selected, can be associated withthe electronic content currently displayed shown in the viewinginterface 310. In this example, the content being illustrated is a menuor catalog of cakes, including depictions of each cake, its title, andits price.

As noted previously, in some implementations, the annotation managementsystem can be configured to identify one or more characters or symbolsin a particular electronic content item as corresponding to a desiredtag. In this specific example, when the user selects or activates the“BUY” tag, the system can be configured to detect all instances of a “$”(dollar sign) with a number, and at each of these instances, pair or addexecutable instructions to the instance. In other implementations, auser can select (e.g., with a mouse click, mouse highlight, or otherinteraction event) each instance where the executable annotation isdesired and manually link or pair the tag with the instance. Once one ormore tags are added or inserted into an electronic content, the sharingor other transference of the content to another location canautomatically trigger the execution of the task, as reflected in FIGS. 4and 5.

In FIG. 4, the first content item 312 has been communicated,transmitted, shared, or sent from the sender to a second user(recipient) via an instant messaging application. In differentimplementations, the augmented electronic content item as presented tothe recipient can offer a ‘call to action’ button or an actionableoption that, if selected, can route the recipient to a pre-defineddestination, receive, store, and/or share information resulting from theselection of the actionable option, and/or guide the recipient through aprocess, as designated by the sender. As the first content item 312 isrendered and displayed for the recipient (here, Martha Johnston) in amessaging interface 410, it can be seen that in addition to originalcontent layer of the electronic content item itself, a second,contextual content layer is also manifested. The contextual contentlayer refers to any content that has been added in response to executionof instructions associated with the selected tag. In this case, thecontextual content layer is presented to the recipient in the form of aplurality of native controls or actionable options (“BUY”) 450 that aredisplayed adjacent to each product. In some implementations, each buttonwhen selected can be configured to perform an action or initiate aprocess. Thus, in some implementations, the ‘task’ to be performed bythe insertion of the annotation by the sender includes rendering of anew (contextual) layer of content along with the original electroniccontent.

In some implementations, the receiver client device interprets theannotated electronic content and the rendering occurs within themessaging application main window or is displayed by a separateapplication (e.g., a web browser invoked with a URL or a map applicationinvoked with an address). In one implementation, the contextual layercan be displayed in various modes, such as overlay (floating above theelectronic content), replace (floating above particular portions of theelectronic content, thereby “hiding” the content), side panel (presentin a side panel adjacent to the electronic content, possibly in the formof a small preview that can be interacted with), and margin (present ina margin area of the electronic content, possibly in the form of a verylimited preview).

If a recipient clicks or otherwise selects one of the actionable options450, as represented by a mouse cursor 452 in FIG. 4, a new process canbe initiated in conjunction with the instant messaging application. Forexample, referring to FIG. 5, a new visual (second content item 514) maybe displayed in the messaging interface 410, providing the recipientwith an opportunity to proceed/confirm with the transaction selected inFIG. 4. Additional actions may occur as the recipient continues toprovide inputs to the system. In FIG. 6, in response to a confirmationby the recipient of this purchase, a webpage 600 has opened on the samecomputing device for processing of the recipient's payment. Thus, it canbe seen that in some implementations, the system can be configured toinitiate or perform processes that involve multiple steps and/orapplications, receive and process inputs from a variety of sources,and/or integrate information occurring at different points in time toprovide a seamless user experience.

As the recipient interacts with the contextual content layer and engagesin any subsequent steps, in different implementations, the system can beoptionally configured to update or notify the sender of the progress orstatus of the transaction. For example, in some implementations, asdepicted in FIG. 7, the instant messaging application can present anautomatic notification 700 informing the sender that a purchase hasoccurred and/or provide details about the transaction. In someimplementations, the sender may also be automatically presented with oneor more suggested responses or options. In this example, a secondinterface 750 is shown within the messaging interface 410 that offers aplurality of options 710 for responding to the notification 700. Inother implementations, the second interface 750 may be independent tothe messaging interface 410, or no additional options may be offered tothe sender.

As noted earlier, in different implementations, the executableannotations can be configured to initiate a wide range of tasks. Anotherexample is presented with reference to FIGS. 8A-8F. In FIG. 8A, a firstuser 810 is shown capturing a photo on a mobile device 812 of awhiteboard 814 that includes a graph 816. In FIG. 8B, the captured image(third content item 820) is viewed on the mobile device 812 and,alongside it, a second annotation interface (“second interface”) 822. Incontrast to the first interface 350 of FIG. 3, the tags offered in thesecond interface 822 are categorized under a “Sharing Tags” header 824,and are directed to tasks that enable sharing of the electronic contentitem. Having chosen an “instant message” option 826 in FIG. 8B, thesender can be prompted to add details to shape the execution of theselected task. In FIG. 8C, the sender has inserted a tag 830 with amessage 834 (“@EricS Should we include this at next week's meeting?”),thereby creating an augmented content item 832. In some implementations,the system can be configured, based at least on the data inputted intotag 830, to initiate a specific set of actions, as will be discussedbelow with reference to FIG. 8F.

In FIG. 8D, the first user opens a social media application 840 andidentifies a recipient (“Contoso Crew”) 842 for the third content item,which in this case corresponds to a larger user group that can includemultiple members. In FIG. 8E, it can be seen that the posting of thethird content item 820 has been successfully added to a recipient grouppage 850. Furthermore, the tag 830 of FIG. 8C is not visible, nor areany indications that the content was augmented revealed to the group. Inother words, in some implementations, the system can be configured toremove some or all traces of the tag 830 when it is shared. In addition,upon sharing of the third content item 820 with the recipient 842, thesystem can interpret the tag content and execute the associated task. Inthis case, as shown in FIG. 8F, the third content item 820 isautomatically brought to the attention of an individual user account 862associated with the “EricS” username via a separate instant message on asecond mobile device 860. Thus, while the third content item 820 wasshared on the recipient group page 850 a result of manual actions by thesender, the subsequent (or substantially simultaneous) presentation ofthe image on Eric Sho's channel 866 as well as any optional messagecontent 868 occurred automatically as a result of the execution ofinstructions conveyed by the tag. Thus, in some implementations, asender can designate or identify one or more individuals that should beparticularly ‘called out’ or notified regarding the shared content item(as well as a specific message content) via the tag, and the system can‘translate’ the tag and automatically convey the message to thedesignated person(s) with the message.

For purposes of clarity, an additional example of the disclosed systemis presented now with reference to FIGS. 9-11. In FIG. 9, a sender(represented by a hand) 900 views a document (fourth content item 910).In this case, the document is an invoice for services. The sender 900would like to receive her manager's approval for the invoice. Indifferent implementations, the system can include provisions forproviding organizations or other users with the ability to custom createor designate tags for day-to-day use. In this example, the sender 900accesses a third annotation interface (“third interface”) 930, whichincludes a header “Shezi Company, Ltd. Tags” 932. The header 932identifies the listed options as being specifically offered for thecompany's use, and/or were created by members of the company inanticipation of the request for tasks commonly requested by employees.In FIG. 9, the sender 900 selects an “Approve Invoice” option 940, whichtriggers an automatic association of a pre-defined tag with the fourthcontent item 910. When the fourth content item 910 is shared withanother user, the tag can be activated and its correspondinginstructions executed. In some implementations, as shown in FIG. 10, thepresentation of the fourth content item 910 can occur within aparticular user interface (here for a messaging application 1010) ordisplay context rendered by receiver's computing device 1000 as a resultof executing the tag associated with the fourth content item 910. Inother words, although the tag itself is not shared, the content of thetag serves to directly transform or reshape the receiver's userexperience of the document. Rather than simply present the document, forexample, as an attachment, the application can present the contentdirectly in the context deemed to be best suited to achieve the sender'sgoals. In some other implementations, the message to the recipient caninclude a portal or link to an Invoice Approval system. When therecipient accesses the link, the document displayed in a way thatimmediately and clearly indicates the type of response desired by thesender, and provides a simple and efficient means of submitting theresponse.

In some implementations, the recipient's selected response can beconveyed back to the sender in the same communication format used by thesender to share the electronic content. In other implementations, thesender can request notification of a response via designation of someother communication mechanism. In some implementations, the executableannotation can be configured with instructions that when executedautomatically update a record or database, or incorporate theinformation from the response into another application. In FIG. 11, itcan be seen that an Invoice Status database 1100 has been automaticallyupdated following submission of the response by the recipient.

As noted earlier, in some implementations, the system may includeprovisions for enhancing the specificity by which users can select orassign particular annotation-task pairs to electronic content items. Insome implementations, users can directly assign a classification, label,or category to one or more desired tasks, and/or designate the type ofannotative tools that should be offered or made available. As oneexample, FIG. 12 presents a first customization menu 1210, in which aplurality of possible tasks 1220 is shown. It should be understood thatthe specific tasks and actions depicted in FIG. 12 are for shownpurposes of simplicity only, and a wide range of other possibleselections (including tasks that are specifically described via manualuser input, such as a series of building blocks or a custom designedpath identifying a particular process that is desired to be triggered).

In FIG. 12, the tasks 1220 include a first task 1222 (“Offer ActionableOptions”) that has been selected by a user, indicating a goal ofaugmenting the electronic content item, such that when the item ispresented to a receiver the presentation will also include a mechanismfor responding efficiently, in a manner that bypasses many cumbersomeuser intensive manual steps that would otherwise be needed to performthe task. For example, this can be conveyed by selectable button(s) oroption(s) that when selected by the recipient will perform somespecified action. In response, in some implementations, the system canpresent a second customization menu 1230, where the task type chosen inthe previous step has resulted in a plurality of actions 1232 beingshown that can be linked to the selectable button, and which will bedisplayed to the recipient. In this example, the user has selected afirst action 1234 (“BUY→PAYMENT”) indicating that activation of theoption by a recipient should trigger a payment process associated withall or a designated portion of the electronic content item. In someimplementations, the user may wish to make use of the payment processworkflow already offered by the system (see first option 1250). However,in other implementations, a user can access another interface that willprovide the user with the opportunity to manually identify and designatethe process that should occur in order to elicit the payment from therecipient (see second option 1252).

In different implementations, the system can offer many types ofannotative task-action pairings. Some examples are also shown in FIG.12, including “Send E-mail” and “Send text message”, the selections ofwhich can indicate that the electronic content item is being annotatedwith a desire of having the system convey the electronic content item toa specific recipient via one or more communication channels. Similarly,“Add to file” can indicate a desire to insert or add the electroniccontent item to an existing or new file located at a specific locationor path. An “Embed in webpage” selection can indicate that theelectronic content item should be inserted or added to a specificwebpage, such as a social media page or blog. Many other tasks arepossible, including but not limited to those providing featuresdescribed earlier with respect to FIGS. 1-11.

In addition, in one embodiment, the manner in which the augmentedcontent is displayed can be configurable from a fixed set of options.Options include, for example, simple formatting, display mode, andautomatic layer launch. Simple formatting includes, for example, textstyle options (e.g., font, color, highlighting, italics, boldface,underlining, and outlining) and indicators (e.g., mini-icons orchiclets) that would be shown to the receiver in conjunction with theparticular electronic content item being shared.

FIG. 13 is a flow chart illustrating an implementation of a method 1300of annotating an electronic content item and initiating a taskassociated with the annotation. A first step 1310 of the method 1300includes presenting, on a first client device, a first user interfacefor a first content viewing application, the first user interfacedisplaying a first electronic content. A second step 1320 includesreceiving, from a first user, a first user input for associating a firsttag with the first electronic content. The first tag represents a firstexecutable process for performing a first task. In a third step 1330,the method includes recording, in response to receiving the first userinput, an association between the first electronic content and the firsttag, and a fourth step 1340 includes receiving, from the first user, asecond user input requesting that the first electronic content betransmitted to a first recipient. A fifth step 1350 includestransmitting, in response to receiving the second user input, the firstelectronic content to the first recipient, and a sixth step 1360involves executing, in response to transmitting the first electroniccontent, the first executable process.

In other implementations, the method may include additional steps oraspects. In some implementations, the method may also includepresenting, on the first client device, an annotation menu offering aplurality of selectable tags that includes the first tag. In some cases,each tag of the plurality of tags represents a different type ofexecutable process that may be associated with electronic content, andthe first user input results from a selection of the first tag from theannotation menu. In another implementation, the method also includespresenting to the first recipient, during execution of the firstexecutable process, a first selectable option for initiating aperformance of an action in conjunction with the first electroniccontent. In some implementations, the action includes providing, to thesecond user, a payment information interface for receiving paymentmethod details, while in other implementations, the action includesmodifying a record associated with the first electronic content.

As another example, the method may include automatically generating,during execution of the first executable process, a first message to asecond recipient, wherein the first message includes a copy of the firstelectronic content. In some cases, transmission to the first recipientoccurs via a first telecommunications service and transmission to thesecond recipient is occurs via a second telecommunications service thatdiffers from the first telecommunications service. As a further example,the method can include receiving, from the first user, a third userinput requesting that a second tag be associated with the firstelectronic content item, the second tag representing a second executableprocess for performing a second task, as well as recording, in responseto receiving the third user input, an association between the firstelectronic content and the second tag. In addition, the method caninclude executing, in response to transmitting the first electroniccontent, the second executable process, and automatically generating, inresponse to executing the second executable process, a second message toa third recipient, the third recipient being a member of the firstrecipient.

In addition, the method can include receiving, from the second user, athird user input selecting the first selectable option, therebytriggering performance of the action, and then presenting, in responseto receiving the third user input, a second selectable option to thefirst user for accessing details associated with the performance of theaction. In one implementation, the method also includes automaticallygenerating a message to the first user indicating that the actionoccurred. In some implementations, the first user input includes atleast a first content element configured to trigger generation of a tag,and the may method further include ascertaining the first user inputincludes the first content element, and then determining, in response toascertaining the first user input includes the first content element,that the first user input corresponds to a request to generate a tag forassociation with the first electronic content. In another example, themethod may involve receiving, from a second user, a third user inputdesignating which selectable tags are to be offered in the annotationmenu. In some cases, a virtual label may be incorporated or added to theelectronic content that is only visible to the first user. In anotherexample, a user may provide an input specifically designating a pairingbetween a tag type, a task, and/or an action.

The use of the disclosed systems and methods can enable users to shareelectronic content in a manner that is better aligned with theirintentions for that item. In addition, organizations can define theirpreferred tags that senders can use as well as define the behavior ofthe listed tags, and how each tag will be manifested at the receiversend. Furthermore the behavior can be defined to manifest in a way thatmakes effective use of the native capabilities at the receiver's end. Indifferent implementations, this system can be configured to interpretvarious augmentation layers and can be capable of retaining, executing,and rendering them on an endpoint at the receivers end. If a receiverinteracts with the contextual layer presented in such a way thattriggers a corresponding process, a service that aggregates theseactions, and triggers follow up workflows based on these interactionscan be initiated.

For the sake of simplicity of description, details are not providedherein for performing various steps. Implementations of the presentdisclosure can make use of any of the features, systems, components,devices, and methods described in U.S. Pat. No. 7,243,301 to Bargeron etal., issued Jul. 10, 2007 and entitled “Common Annotation Framework,”the disclosure of which is herein incorporated by reference in itsentirety.

The detailed examples of systems, devices, and techniques described inconnection with FIGS. 1-13 are presented herein for illustration of thedisclosure and its benefits. Such examples of use should not beconstrued to be limitations on the logical process implementations ofthe disclosure, nor should variations of user interface methods fromthose described herein be considered outside the scope of the presentdisclosure. In some implementations, various features described in FIGS.1-13 are implemented in respective modules, which may also be referredto as, and/or include, logic, components, units, and/or mechanisms.Modules may constitute either software modules (for example, codeembodied on a machine-readable medium) or hardware modules.

In some examples, a hardware module may be implemented mechanically,electronically, or with any suitable combination thereof. For example, ahardware module may include dedicated circuitry or logic that isconfigured to perform certain operations. For example, a hardware modulemay include a special-purpose processor, such as a field-programmablegate array (FPGA) or an Application Specific Integrated Circuit (ASIC).A hardware module may also include programmable logic or circuitry thatis temporarily configured by software to perform certain operations, andmay include a portion of machine-readable medium data and/orinstructions for such configuration. For example, a hardware module mayinclude software encompassed within a programmable processor configuredto execute a set of software instructions. It will be appreciated thatthe decision to implement a hardware module mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (for example, configured by software) may be driven by cost,time, support, and engineering considerations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity capable of performing certain operations andmay be configured or arranged in a certain physical manner, be that anentity that is physically constructed, permanently configured (forexample, hardwired), and/or temporarily configured (for example,programmed) to operate in a certain manner or to perform certainoperations described herein. As used herein, “hardware-implementedmodule” refers to a hardware module. Considering examples in whichhardware modules are temporarily configured (for example, programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where a hardware module includesa programmable processor configured by software to become aspecial-purpose processor, the programmable processor may be configuredas respectively different special-purpose processors (for example,including different hardware modules) at different times. Software mayaccordingly configure a particular processor or processors, for example,to constitute a particular hardware module at one instance of time andto constitute a different hardware module at a different instance oftime. A hardware module implemented using one or more processors may bereferred to as being “processor implemented” or “computer implemented.”

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (for example, over appropriate circuits andbuses) between or among two or more of the hardware modules. Inimplementations in which multiple hardware modules are configured orinstantiated at different times, communications between such hardwaremodules may be achieved, for example, through the storage and retrievalof information in memory devices to which the multiple hardware moduleshave access. For example, one hardware module may perform an operationand store the output in a memory device, and another hardware module maythen access the memory device to retrieve and process the stored output.

In some examples, at least some of the operations of a method may beperformed by one or more processors or processor-implemented modules.Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by, and/or among, multiplecomputers (as examples of machines including processors), with theseoperations being accessible via a network (for example, the Internet)and/or via one or more software interfaces (for example, an applicationprogram interface (API)). The performance of certain of the operationsmay be distributed among the processors, not only residing within asingle machine, but deployed across a number of machines. Processors orprocessor-implemented modules may be located in a single geographiclocation (for example, within a home or office environment, or a serverfarm), or may be distributed across multiple geographic locations.

FIG. 14 is a block diagram 1400 illustrating an example softwarearchitecture 1402, various portions of which may be used in conjunctionwith various hardware architectures herein described, which mayimplement any of the above-described features. FIG. 14 is a non-limitingexample of a software architecture and it will be appreciated that manyother architectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 1402 may execute on hardwaresuch as a device 150 of FIG. 1A that includes, among other things,document storage 1070, processors, memory, and input/output (I/O)components. A representative hardware layer 1404 is illustrated and canrepresent, for example, the device 150 of FIG. 1. The representativehardware layer 1404 includes a processing unit 1406 and associatedexecutable instructions 1408. The executable instructions 1408 representexecutable instructions of the software architecture 1402, includingimplementation of the methods, modules and so forth described herein.The hardware layer 1404 also includes a memory/storage 1410, which alsoincludes the executable instructions 1408 and accompanying data. Thehardware layer 1404 may also include other hardware modules 1412.Instructions 1408 held by processing unit 1408 may be portions ofinstructions 1408 held by the memory/storage 1410.

The example software architecture 1402 may be conceptualized as layers,each providing various functionality. For example, the softwarearchitecture 1402 may include layers and components such as an operatingsystem (OS) 1414, libraries 1416, frameworks 1418, applications 1420,and a presentation layer 1444. Operationally, the applications 1420and/or other components within the layers may invoke API calls 1424 toother layers and receive corresponding results 1426. The layersillustrated are representative in nature and other softwarearchitectures may include additional or different layers. For example,some mobile or special purpose operating systems may not provide theframeworks/middleware 1418.

The OS 1414 may manage hardware resources and provide common services.The OS 1414 may include, for example, a kernel 1428, services 1430, anddrivers 1432. The kernel 1428 may act as an abstraction layer betweenthe hardware layer 1404 and other software layers. For example, thekernel 1428 may be responsible for memory management, processormanagement (for example, scheduling), component management, networking,security settings, and so on. The services 1430 may provide other commonservices for the other software layers. The drivers 1432 may beresponsible for controlling or interfacing with the underlying hardwarelayer 1404. For instance, the drivers 1432 may include display drivers,camera drivers, memory/storage drivers, peripheral device drivers (forexample, via Universal Serial Bus (USB)), network and/or wirelesscommunication drivers, audio drivers, and so forth depending on thehardware and/or software configuration.

The libraries 1416 may provide a common infrastructure that may be usedby the applications 1420 and/or other components and/or layers. Thelibraries 1416 typically provide functionality for use by other softwaremodules to perform tasks, rather than rather than interacting directlywith the OS 1414. The libraries 1416 may include system libraries 1434(for example, C standard library) that may provide functions such asmemory allocation, string manipulation, file operations. In addition,the libraries 1416 may include API libraries 1436 such as medialibraries (for example, supporting presentation and manipulation ofimage, sound, and/or video data formats), graphics libraries (forexample, an OpenGL library for rendering 2D and 3D graphics on adisplay), database libraries (for example, SQLite or other relationaldatabase functions), and web libraries (for example, WebKit that mayprovide web browsing functionality). The libraries 1416 may also includea wide variety of other libraries 1438 to provide many functions forapplications 1420 and other software modules.

The frameworks 1418 (also sometimes referred to as middleware) provide ahigher-level common infrastructure that may be used by the applications1420 and/or other software modules. For example, the frameworks 1418 mayprovide various graphic user interface (GUI) functions, high-levelresource management, or high-level location services. The frameworks1418 may provide a broad spectrum of other APIs for applications 1420and/or other software modules.

The applications 1420 include built-in applications 1440 and/orthird-party applications 1442. Examples of built-in applications 1440may include, but are not limited to, a contacts application, a browserapplication, a location application, a media application, a messagingapplication, and/or a game application. Third-party applications 1442may include any applications developed by an entity other than thevendor of the particular platform. The applications 1420 may usefunctions available via OS 1414, libraries 1416, frameworks 1418, andpresentation layer 1444 to create user interfaces to interact withusers.

Some software architectures use virtual machines, as illustrated by avirtual machine 1448. The virtual machine 1448 provides an executionenvironment where applications/modules can execute as if they wereexecuting on a hardware machine (such as the machine 1000 of FIG. 10,for example). The virtual machine 1448 may be hosted by a host OS (forexample, OS 1414) or hypervisor, and may have a virtual machine monitor1446 which manages operation of the virtual machine 1448 andinteroperation with the host operating system. A software architecture,which may be different from software architecture 1402 outside of thevirtual machine, executes within the virtual machine 1448 such as an OS1450, libraries 1452, frameworks 1454, applications 1456, and/or apresentation layer 1458.

FIG. 15 is a block diagram illustrating components of an example machine1500 configured to read instructions from a machine-readable medium (forexample, a machine-readable storage medium) and perform any of thefeatures described herein. The example machine 1500 is in a form of acomputer system, within which instructions 1516 (for example, in theform of software components) for causing the machine 1500 to perform anyof the features described herein may be executed. As such, theinstructions 1516 may be used to implement modules or componentsdescribed herein. The instructions 1516 cause unprogrammed and/orunconfigured machine 1500 to operate as a particular machine configuredto carry out the described features. The machine 1500 may be configuredto operate as a standalone device or may be coupled (for example,networked) to other machines. In a networked deployment, the machine1500 may operate in the capacity of a server machine or a client machinein a server-client network environment, or as a node in a peer-to-peeror distributed network environment. Machine 1500 may be embodied as, forexample, a server computer, a client computer, a personal computer (PC),a tablet computer, a laptop computer, a netbook, a set-top box (STB), agaming and/or entertainment system, a smart phone, a mobile device, awearable device (for example, a smart watch), and an Internet of Things(IoT) device. Further, although only a single machine 1500 isillustrated, the term “machine” includes a collection of machines thatindividually or jointly execute the instructions 1516.

The machine 1500 may include processors 1510, memory 1530, and I/Ocomponents 1550, which may be communicatively coupled via, for example,a bus 1502. The bus 1502 may include multiple buses coupling variouselements of machine 1500 via various bus technologies and protocols. Inan example, the processors 1510 (including, for example, a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), an ASIC, or a suitable combination thereof) mayinclude one or more processors 1512 a to 1512 n that may execute theinstructions 1516 and process data. In some examples, one or moreprocessors 1510 may execute instructions provided or identified by oneor more other processors 1510. The term “processor” includes amulti-core processor including cores that may execute instructionscontemporaneously. Although FIG. 15 shows multiple processors, themachine 1500 may include a single processor with a single core, a singleprocessor with multiple cores (for example, a multi-core processor),multiple processors each with a single core, multiple processors eachwith multiple cores, or any combination thereof. In some examples, themachine 1500 may include multiple processors distributed among multiplemachines.

The memory/storage 1530 may include a main memory 1532, a static memory1534, or other memory, and a storage unit 1536, both accessible to theprocessors 1510 such as via the bus 1502. The storage unit 1536 andmemory 1532, 1534 store instructions 1516 embodying any one or more ofthe functions described herein. The memory/storage 1530 may also storetemporary, intermediate, and/or long-term data for processors 1510. Theinstructions 1516 may also reside, completely or partially, within thememory 1532, 1534, within the storage unit 1536, within at least one ofthe processors 1510 (for example, within a command buffer or cachememory), within memory at least one of I/O components 1550, or anysuitable combination thereof, during execution thereof. Accordingly, thememory 1532, 1534, the storage unit 1536, memory in processors 1510, andmemory in I/O components 1550 are examples of machine-readable media.

As used herein, “machine-readable medium” refers to a device able totemporarily or permanently store instructions and data that causemachine 1500 to operate in a specific fashion. The term“machine-readable medium,” as used herein, does not encompass transitoryelectrical or electromagnetic signals per se (such as on a carrier wavepropagating through a medium); the term “machine-readable medium” maytherefore be considered tangible and non-transitory. Non-limitingexamples of a non-transitory, tangible machine-readable medium mayinclude, but are not limited to, nonvolatile memory (such as flashmemory or read-only memory (ROM)), volatile memory (such as a staticrandom-access memory (RAM) or a dynamic RAM), buffer memory, cachememory, optical storage media, magnetic storage media and devices,network-accessible or cloud storage, other types of storage, and/or anysuitable combination thereof. The term “machine-readable medium” appliesto a single medium, or combination of multiple media, used to storeinstructions (for example, instructions 1516) for execution by a machine1500 such that the instructions, when executed by one or more processors1510 of the machine 1500, cause the machine 1500 to perform and one ormore of the features described herein. Accordingly, a “machine-readablemedium” may refer to a single storage device, as well as “cloud-based”storage systems or storage networks that include multiple storageapparatus or devices.

The I/O components 1550 may include a wide variety of hardwarecomponents adapted to receive input, provide output, produce output,transmit information, exchange information, capture measurements, and soon. The specific I/O components 1550 included in a particular machinewill depend on the type and/or function of the machine. For example,mobile devices such as mobile phones may include a touch input device,whereas a headless server or IoT device may not include such a touchinput device. The particular examples of I/O components illustrated inFIG. 15 are in no way limiting, and other types of components may beincluded in machine 1500. The grouping of I/O components 1550 are merelyfor simplifying this discussion, and the grouping is in no way limiting.In various examples, the I/O components 1550 may include user outputcomponents 1552 and user input components 1554. User output components1552 may include, for example, display components for displayinginformation (for example, a liquid crystal display (LCD) or aprojector), acoustic components (for example, speakers), hapticcomponents (for example, a vibratory motor or force-feedback device),and/or other signal generators. User input components 1554 may include,for example, alphanumeric input components (for example, a keyboard or atouch screen), pointing components (for example, a mouse device, atouchpad, or another pointing instrument), and/or tactile inputcomponents (for example, a physical button or a touch screen thatprovides location and/or force of touches or touch gestures) configuredfor receiving various user inputs, such as user commands and/orselections.

In some examples, the I/O components 1550 may include biometriccomponents 1556 and/or position components 1562, among a wide array ofother environmental sensor components. The biometric components 1556 mayinclude, for example, components to detect body expressions (forexample, facial expressions, vocal expressions, hand or body gestures,or eye tracking), measure biosignals (for example, heart rate or brainwaves), and identify a person (for example, via voice-, retina-, and/orfacial-based identification). The position components 1562 may include,for example, location sensors (for example, a Global Position System(GPS) receiver), altitude sensors (for example, an air pressure sensorfrom which altitude may be derived), and/or orientation sensors (forexample, magnetometers).

The I/O components 1550 may include communication components 1564,implementing a wide variety of technologies operable to couple themachine 1500 to network(s) 1570 and/or device(s) 1580 via respectivecommunicative couplings 1572 and 1582. The communication components 1564may include one or more network interface components or other suitabledevices to interface with the network(s) 1570. The communicationcomponents 1564 may include, for example, components adapted to providewired communication, wireless communication, cellular communication,Near Field Communication (NFC), Bluetooth communication, Wi-Fi, and/orcommunication via other modalities. The device(s) 1580 may include othermachines or various peripheral devices (for example, coupled via USB).

In some examples, the communication components 1564 may detectidentifiers or include components adapted to detect identifiers. Forexample, the communication components 1564 may include Radio FrequencyIdentification (RFID) tag readers, NFC detectors, optical sensors (forexample, one- or multi-dimensional bar codes, or other optical codes),and/or acoustic detectors (for example, microphones to identify taggedaudio signals). In some examples, location information may be determinedbased on information from the communication components 1562, such as,but not limited to, geo-location via Internet Protocol (IP) address,location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless stationidentification and/or signal triangulation.

While various implementations have been described, the description isintended to be exemplary, rather than limiting, and it is understoodthat many more implementations and implementations are possible that arewithin the scope of the implementations. Although many possiblecombinations of features are shown in the accompanying figures anddiscussed in this detailed description, many other combinations of thedisclosed features are possible. Any feature of any implementation maybe used in combination with or substituted for any other feature orelement in any other implementation unless specifically restricted.Therefore, it will be understood that any of the features shown and/ordiscussed in the present disclosure may be implemented together in anysuitable combination. Accordingly, the implementations are not to berestricted except in light of the attached claims and their equivalents.Also, various modifications and changes may be made within the scope ofthe attached claims.

While the foregoing has described what are considered to be the bestmode and/or other examples, it is understood that various modificationsmay be made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that the teachings may beapplied in numerous applications, only some of which have been describedherein. It is intended by the following claims to claim any and allapplications, modifications and variations that fall within the truescope of the present teachings.

Unless otherwise stated, all measurements, values, ratings, positions,magnitudes, sizes, and other specifications that are set forth in thisspecification, including in the claims that follow, are approximate, notexact. They are intended to have a reasonable range that is consistentwith the functions to which they relate and with what is customary inthe art to which they pertain.

The scope of protection is limited solely by the claims that now follow.That scope is intended and should be interpreted to be as broad as isconsistent with the ordinary meaning of the language that is used in theclaims when interpreted in light of this specification and theprosecution history that follows and to encompass all structural andfunctional equivalents. Notwithstanding, none of the claims are intendedto embrace subject matter that fails to satisfy the requirement ofSections 101, 102, or 103 of the Patent Act, nor should they beinterpreted in such a way. Any unintended embracement of such subjectmatter is hereby disclaimed.

Except as stated immediately above, nothing that has been stated orillustrated is intended or should be interpreted to cause a dedicationof any component, step, feature, object, benefit, advantage, orequivalent to the public, regardless of whether it is or is not recitedin the claims.

It will be understood that the terms and expressions used herein havethe ordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”or any other variation thereof, are intended to cover a non-exclusiveinclusion, such that a process, method, article, or apparatus thatcomprises a list of elements does not include only those elements butmay include other elements not expressly listed or inherent to suchprocess, method, article, or apparatus. An element proceeded by “a” or“an” does not, without further constraints, preclude the existence ofadditional identical elements in the process, method, article, orapparatus that comprises the element.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various examples for the purpose of streamlining thedisclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claims require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed example. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separately claimed subject matter.

What is claimed is:
 1. A data processing system comprising: at least oneprocessor; and one or more computer readable media includinginstructions which, when executed by the at least one processor, causethe at least one processor to: provide, to a first client device over acommunication network, a plurality of tags and a plurality of tasks,wherein each tag is configured to label electronic content, wherein eachtag is associated with a respective one of the plurality of tasksrepresenting an executable process to be performed on the electroniccontent responsive to that tag being associated with the electroniccontent, and wherein the plurality of tags includes a first tagassociated with a first task of the plurality of tasks; receive over thecommunication network, from the first client device, a request toassociate the first tag with first electronic content; record, inresponse to receiving the request, an association between the firstelectronic content and the first tag; and transmit over thecommunication network to a first recipient device: the first electroniccontent, and an instruction to the first recipient device to cause thefirst recipient device to execute, in response to receiving the firstelectronic content, the first task on the first electronic content. 2.The system of claim 1, wherein the instructions further cause the atleast one processor to transmit over the communication network, to thefirst client device, an instruction to cause the first client device todisplay an annotation menu including the first tag among the pluralityof tags, and the request results from a selection of the first tag fromthe annotation menu.
 3. The system of claim 1, wherein the firstexecutable process causes the first recipient device to display a firstselectable option for initiating a performance of the first task inconjunction with the first electronic content.
 4. The system of claim 3,wherein the action includes causing the first recipient device todisplay a payment information interface for receiving payment methoddetails.
 5. The system of claim 3, wherein the action includes modifyinga record that stores information about a status of the first electroniccontent.
 6. The system of claim 1, wherein the instructions furthercause the at least one processor to automatically generate, duringexecution of the first executable process, a first message to a secondrecipient device that differs from the first recipient device, whereinthe first message includes a copy of the first electronic content. 7.The system of claim 6, wherein transmission to the first recipientoccurs via a first communication network and transmission to the secondrecipient occurs via a second communication network that differs fromthe first communication network.
 8. The system of claim 2, wherein theinstructions further cause the at least one processor to: receive, fromthe first client device, a second request that a second tag beassociated with the first electronic content item, the second tagrepresenting a second executable process for performing a second task;record, in response to receiving the second request, a secondassociation between the first electronic content and the second tag;transmit an instruction to the first recipient device to cause the firstrecipient device to execute, in response to receiving the firstelectronic content, the second executable process on the firstelectronic content; and automatically generate, in response totransmitting the instruction to cause the first recipient device toexecute the second executable process, a second message to a thirdrecipient, the third recipient being a member of the first recipient. 9.The system of claim 3, wherein the instructions further cause the atleast one processor to: receive, from a second client device, a secondrequest selecting the first selectable option, thereby triggeringperformance of the first task; and transmit, in response to receivingthe second request, an instruction to the first recipient device tocause the first recipient device to display a second selectable optionfor accessing details associated with the performance of the first task.10. The system of claim 1, wherein the instructions further cause the atleast one processor to automatically generate a message to the firstclient device indicating that the action occurred.
 11. A method executedon one or more computing devices, the method comprising: providing, to afirst client device over a communication network, a plurality of tagsand a plurality of tasks, wherein each tag is configured to labelelectronic content, wherein each tag is associated with a respective oneof the plurality of tasks representing an executable process to beperformed on the electronic content responsive to that tag beingassociated with the electronic content, and wherein the plurality oftags includes a first tag associated with a first task of the pluralityof tasks; receiving over the communication network, from the firstclient device, a request to associate the first tag with firstelectronic content; recording, in response to receiving the request, anassociation between the first electronic content and the first tag;transmitting over the communication network to a first recipient device:the first electronic content, and an instruction to the first recipientdevice to cause the first recipient device to execute, in response toreceiving the first electronic content, the first task on the firstelectronic content.
 12. The method of claim 11, further comprisingtransmitting over the communication network, to the first client device,an instruction to cause the first client device to display an annotationmenu including the first tag among the plurality of tags, and the firstrequest from a selection of the first tag from the annotation menu. 13.The method of claim 11, further comprising presenting on the firstrecipient device, during execution of the first executable process, afirst selectable option for initiating a performance of an action inconjunction with the first electronic content.
 14. The method of claim13, wherein the action includes causing the first recipient device todisplay a payment information interface for receiving payment methoddetails.
 15. The method of claim 13, wherein the action includesmodifying a record that stores information about a status of the firstelectronic content.
 16. The method of claim 11, further comprisingautomatically generating, during execution of the first executableprocess, a first message to a second recipient device that differs fromthe first recipient device, wherein the first message includes a copy ofthe first electronic content.
 17. The method of claim 16, whereintransmission to the first recipient occurs via a first communicationnetwork and transmission to the second recipient occurs via a secondcommunication network that differs from the first communication network.18. The method of claim 12, further comprising: receiving, from thefirst client device, a second request that a second tag be associatedwith the first electronic content item, the second tag representing asecond executable process for performing a second task; recording, inresponse to receiving the second request, a second association betweenthe first electronic content and the second tag; transmitting aninstruction to the first recipient device to cause the first recipientdevice to execute, in response to receiving the first electroniccontent, the second executable process on the first electronic content;and automatically generating, in response to transmitting theinstruction to cause the first recipient device to execute the secondexecutable process, a second message to a third recipient, the thirdrecipient being a member of the first recipient.
 19. The method of claim13, further comprising: receiving, from a second client device, a secondrequest selecting the first selectable option, thereby triggeringperformance of the first task; and transmitting, in response toreceiving the second request, an instruction to the first recipientdevice to cause the first recipient device to display a secondselectable option to the first user for accessing details associatedwith the performance of the action.
 20. A computer-readable storagemedium on which are stored instructions that, when executed, cause aprocessor of a programmable device to perform functions of: providing,to a first client device over a communication network, a plurality oftags and a plurality of tasks, wherein each tag is configured to labelelectronic content, wherein each tag is associated with a respective oneof the plurality of tasks representing an executable process to beperformed on the electronic content responsive to that tag beingassociated with the electronic content, and wherein the plurality oftags includes a first tag associated with a first task of the pluralityof tasks; receiving over the communication network, from the firstclient device, a request to associate the first tag with firstelectronic content; recording, in response to receiving the request, anassociation between the first electronic content and the first tag;transmitting over the communication network to a first recipient device:the first electronic content, and an instruction to the first recipientdevice to cause the first recipient device to execute, in response toreceiving the first electronic content, the first task on the firstelectronic content.